%\documentclass[twoside,titlepage]{report}
%\usepackage{fullpage}
%\usepackage{graphicx}
%\usepackage{verbatim}
%\usepackage{wrapfig}
%\usepackage{amsmath}
%\usepackage{moreverb}
%\newcommand{\loadsrc}[1]{
%\subsection{#1}
%{\scriptsize{
%\verbatimtabinput[8]{#1}}}
%}

%
%\begin{document}


\section{Test Plan}
All tests can be executed with the command \texttt{make test}.  The testing framework includes a Makefile, a python script, and a set of MatPix source files (extension .mpx).  The two main scripts are: \\ \\

\begin{tabular}{|c|c|c|}
\hline
\textbf{File} & \textbf{Description} & \textbf{Authors} \\
\hline
Makefile & compile MatPix and C++ code, execute run.py & Ollie, Rob\\
run.py & compare error logs, execute binaries, compare results & Ollie, Rob\\
\hline
\end{tabular}
\subsection{Test Suite}


The set of tests we used to test our language checks functionality and errors.  Errors could occur in the lexer, parser, tree walker, or at runtime.  We will consider lexer, parser, and tree walker errors to be compile time errors.

\subsubsection{Successful Tests}

The tests for functionality are: \\ \\

\begin{tabular}{|c|c|c|}
\hline
\textbf{File} & \textbf{Description} & \textbf{Authors} \\
\hline
arithmetic.mpx & basic arithmetic & Ben \\
constmatrix.mpx & create a constant matrix & Ollie \\
ctrlFlow.mpx & types of control flow - if, else, while, for & Ben \\
dotprod.mpx & dot product matrix operations & Ollie \\
functiondef.mpx & function definitions & Rob \\
gauss\_jordan.mpx & gauss jordan algorithm & David \\
logictest.mpx & boolean operations & Ben \\
recursive.mpx & recursive factorial function & Rob \\
regression.mpx & regression & David \\
sliceAssign.mpx & assign to a slice of a matrix & Ben \\
sum.mpx & sum of vector using dot product & David \\
transpose.mpx & transpose a matrix & David \\
\hline
\end{tabular} \\ \\

\noindent
These tests represent all of the functionality of MatPix.  We test scalar and matrix basic mathematical operations, function definitions and calls, all types of control flow, and assignment to a slice or portion of a matrix.  We also demonstrate that one can write advanced algorithms that require linear algebra such as linear regression in MatPix.

\subsubsection{Compiler Error Tests}

The following tests cause the MatPix-ANTLR compiler to fail, rightfully: \\ \\

\begin{tabular}{|c|c|c|}
\hline
\textbf{File} & \textbf{Description} & \textbf{Authors} \\
\hline
lvalue\_fail\_m.mpx & invalid string to the left of an assignment & Rob \\
scope\_if\_fail\_m.mpx & identifier is out of scope in an if statement & Ben \\
scope\_func1\_fail\_m.mpx & Out of scope ID used in function & Rob \\
scope\_func2\_fail\_m.mpx & ID defined in function is used in main & Rob \\
funcCall\_notDef\_fail\_m.mpx & Undefined function is called & Rob \\
funcDef\_dup\_fail\_m.mpx & Two functions defined with same signature & Rob \\
syntax\_fail\_m.mpx & Some syntax errors & Rob \\
matIndex\_fail\_m.mpx & Indexes a matrix before it is created & David \\
\hline
\end{tabular} \\ \\

The above tests were chosen to test all the places the compiler could fail.  We allow all expressions to appear on the left hand side of an assignment in the parser, so we catch invalid left-values  as an error in the tree walker.  Scope functionality is handled uniformly for all new scopes (function definitions, control flows, stand alone blocks of code); thus, it is sufficient to test that scope operates as expected in any one of these types of blocks.  Syntax errors are caught by the parser and/or lexer and are reported.  If any error is found during MatPix compilation, the c++ file is not generated.

\subsubsection{Runtime Error Tests}

The following tests fail at runtime:\\ \\

\begin{tabular}{|c|c|c|}
\hline
\textbf{File} & \textbf{Description} & \textbf{Authors} \\
\hline
badDim\_add\_fail\_r.mpx & dimensions do not match in addition operation & Rob \\
badDim\_dotprod\_fail\_r.mpx & Inner dimensions do not match in dot product operation & David \\
range\_inv\_row\_fail\_r.mpx & Invalid range from high to low in row & Rob \\
range\_inv\_col\_fail\_r.mpx & Invalid range from high to low in column & Rob \\
range\_oob\_col\_fail\_r.mpx & Range out of bounds in column slicing & Ben \\
range\_oob\_row\_fail\_r.mpx  & Range out of bounds in row slicing & Ben \\
range\_neg\_fail\_r.mpx & Range is negative & Ben \\
\hline
\end{tabular} \\ \\

The above tests were chosen to test anywhere we expect a runtime error might occur.  Since we allow matrices to be redefined with different sizes, we must verify matrix dimensions at runtime.  We need to check matrix dimensions in mathematical operations and when indexing a matrix.  When indexing a matrix, the ranges must be within range and in increasing order.  That is, the range 1:3 is valid but the range 3:1 is not.

\subsection{Test Automation}

Testing is completely automated and is handled by \texttt{Makefile} and \texttt{run.py}.  \texttt{Makefile} follows:
\subsubsection{Makefile}
\verbatimtabinput{../LRM/tests/Makefile}

The \texttt{Makefile} compiles all the code and stores error logs aside for later comparison.  Executing \texttt{make test} causes all the code to be compiled and executed via \texttt{run.py}.  The python script tests the result of each test listed above: successful tests, MatPix-ANTLR compiler errors, and runtime errors.  Each test result's error and standard output is compared with expected error and standard output for that test.  The expected output for each test is stored in \texttt{bin/pass/}.  If a test has the suffix \texttt{\_fail\_m.mpx} then its ``pass log" will be the standard error reported by the MatPix-ANTLR compiler.  If a test has the suffix \texttt{\_fail\_r.mpx} then its pass log is the standard error reported upon execution.  At the end of each test, if the pass log and the expected output are not identical, an error is reported for that test.
\subsubsection{run.py}
\verbatimtabinput{../LRM/tests/bin/run.py}

{\scriptsize{
\subsection{Example tests}
Three tests that demonstrate a good amount of our language are Sum, Gauss Jordan, and Slice Assign.
\subsubsection{Sum: tests/mpx/sum.mpx}
The MatPix code for Sum follows.
\verbatimtabinput{../LRM/tests/mpx/sum.mpx}
\subsubsection{Sum: tests/cpp/sum.cpp}
The generated C++ code for Sum follows.
\verbatimtabinput{../LRM/tests/cpp/sum.cpp}
\subsubsection{Sum: tests/bin/pass/sum.log}
The output of Sum follows.
\verbatimtabinput{../LRM/tests/bin/pass/sum.log}

\subsubsection{Gauss Jordan: tests/mpx/gauss\_jordan.mpx}
The MatPix code for Gauss Jordan follows.
\verbatimtabinput{../LRM/tests/mpx/gauss_jordan.mpx}
\subsubsection{Gauss Jordan: tests/cpp/gauss\_jordan.cpp}
The generated C++ code for Gauss Jordan follows.
\verbatimtabinput{../LRM/tests/cpp/gauss_jordan.cpp}
\subsubsection{Gauss Jordan: tests/bin/pass/gauss\_jordan.log}
The output of Gauss Jordan follows.
\verbatimtabinput{../LRM/tests/bin/pass/gauss_jordan_clean.log}
\subsubsection{Slice Assign: tests/mpx/sliceAssign.mpx}

The MatPix code for Slice Assign follows.
\verbatimtabinput{../LRM/tests/mpx/sliceAssign.mpx}
\subsubsection{Slice Assign: tests/cpp/sliceAssign.cpp}
The generated C++ code for Slice Assign follows.
\verbatimtabinput{../LRM/tests/cpp/sliceAssign.cpp}
\subsubsection{Slice Assign: tests/bin/pass/sliceAssign.log}
The output of Slice Assign follows.
\verbatimtabinput{../LRM/tests/bin/pass/sliceAssign.log}
}}
%\end{document}
